Method and apparatus for utility-based dynamic resource allocation in a distributed computing system

ABSTRACT

In one embodiment, the present invention is a method for allocation of finite computational resources amongst multiple entities, wherein the method is structured to optimize the business value of an enterprise providing computational services. One embodiment of the inventive method involves establishing, for each entity, a service level utility indicative of how much business value is obtained for a given level of computational system performance. The service-level utility for each entity is transformed into a corresponding resource-level utility indicative of how much business value may be obtained for a given set or amount of resources allocated to the entity. The resource-level utilities for each entity are aggregated, and new resource allocations are determined and executed based upon the resource-level utility information. The invention is thereby capable of making rapid allocation decisions, according to time-varying need or value of the resources by each of the entities.

BACKGROUND

The present invention relates generally to data processing systems, andrelates more particularly to the management of hardware and softwarecomponents of data processing systems. Specifically, the presentinvention provides a method and apparatus for automatic allocation ofcomputing resources amongst multiple entities that obtain value byutilizing the resources to perform computation.

The problem of how to optimally allocate a limited set of resourcesamongst multiple entities that use or consume the resources has beenextensively studied in disciplines including economics, manufacturing,telecommunications networks, and computing systems. Within the latterdomain, the recent evolution of highly interconnected, rapidly changing,distributed computing systems such as the Internet has made itincreasingly important to be able to rapidly compute and executeresource allocation decisions in an automated fashion.

Traditional approaches to provisioning and capacity planning typicallyaim to achieve an external value of some overall system performancemetric (e.g., maximum average throughput or minimum average responsetime). Other conventional techniques employ market-based mechanisms forresource allocation (e.g., auction bidding or bilateral negotiationmechanisms). For example, a commonly used approach has been toanticipate the maximum possible load on the system, and then performone-time static allocation of resources capable of handling the maximumload within a specified margin of safety. A common problem with suchapproaches is that, with modern workloads such as hit rates on Webpages, the demand rate may vary dynamically and rapidly over many ordersof magnitude, and a system that is statically provisioned for its peakworkload may spend nearly all its time sitting idle.

Thus, there is a need in the art for a method and apparatus for dynamicresource allocation in distributed computing systems.

SUMMARY OF THE INVENTION

In one embodiment, the present invention is a method for optimal andautomatic allocation of finite resources (e.g., hardware or softwarethat can be used within any overall process that performs computation)amongst multiple entities that can provide computational services giventhe resource(s). One embodiment of the inventive method involvesestablishing, for each entity, a service level utility indicative of howmuch business value is obtained for a given level of computationalsystem performance and for a given level of demand for computingservice. Each entity is capable of transforming its respectiveservice-level utility into a corresponding resource-level utilityindicative of how much business value may be obtained for a given set oramount of resources allocated to the entity. The resource-levelutilities for each entity are aggregated, and resource allocations aresubsequently determined and executed based upon the dynamicresource-level utility information established. The invention is therebycapable of making rapid allocation decisions, according to time-varyingneed or value of the resources by each of the entities. In addition, theinventive method is motivated by the perspective of an enterprisecomprising multiple entities that use said finite computationalresources to provide service to one or more customers, and is thusstructured to optimize the business value of the enterprise.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited embodiments of theinvention are attained and can be understood in detail, a moreparticular description of the invention, briefly summarized above, maybe obtained by reference to the embodiments thereof which areillustrated in the appended drawings. It is to be noted, however, thatthe appended drawings illustrate only typical embodiments of thisinvention and are therefore not to be considered limiting of its scope,for the invention may admit to other equally effective embodiments.

FIG. 1 is a diagram of a networked data processing system in which thepresent invention may be implemented;

FIG. 2 is an overall view of a resource allocation system in accordancewith one embodiment of the present invention;

FIG. 3 is a flow chart illustrating one embodiment of a method fordynamically allocating resources among multiple applicationenvironments;

FIG. 4 is a diagram illustrating the detailed functionality of anapplication environment module which constitutes a component of theoverall system shown in FIG. 2; and

FIG. 5 is a high level block diagram of the present inventionimplemented using a general purpose computing device.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one embodiment, the present invention is a method for optimal andautomatic allocation of finite resources amongst multiple entities thatcan perform computational work given the resource(s). For the purposesof the present invention, the term “resource” may indicate an entirehardware or software component (e.g., a compute server, a storagedevice, a RAM circuit or a database server), or a portion of a component(e.g., bandwidth access or a fraction of a server). The method may beimplemented, for example, within a data processing system such as anetwork, a server, or a client computer. The invention is capable ofmaking allocation decisions in real time, according to time-varying needor value of the resources by each of the entities, thereby resolving theshortcomings associated with typical static resource allocationtechniques. In addition, the method is structured to optimize thebusiness value of an enterprise that provides computing services tomultiple entities using said finite computational resources.

FIG. 1 is a schematic illustration of one embodiment of a network dataprocessing system 100 comprising a network of computers (e.g., clients)in which the present invention may be implemented. The network dataprocessing system 100 includes a network 102, a server 104, a storageunit 106 and a plurality of clients 108, 110 and 112. The network 102 isthe medium used to provide communications links between the server 104,storage unit 106 and clients 108, 110, 112 connected together withinnetwork data processing system 100. The network 102 may includeconnections, such as wire, wireless communication links, or fiber opticcables.

In the embodiment illustrated, the server 104 provides data, such asboot files, operating system images, and applications to the clients108, 110, 112 (i.e., the clients 108, 110, and 112 are clients to server104). The clients 108, 110, and 112 may be, for example, personalcomputers or network computers. Although the network data processingsystem 100 depicted in FIG. 1 comprises a single server 104 and threeclients, 108, 100, 112, those skilled in the art will recognize that thenetwork data processing system 100 may include additional servers,clients, and other devices not shown in FIG. 1.

In one embodiment, the network data processing system 100 is theInternet, with the network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. In further embodiments, the network data processingsystem 100 is implemented as an intranet, a local area network (LAN), ora wide area network (WAN). Furthermore, although FIG. 1 illustrates anetwork data processing system 100 in which the method of the presentinvention my be implemented, those skilled in the art will realize thatthe present invention may be implemented in a variety of other dataprocessing systems, including servers (e.g., server 104) and clientcomputers (e.g., clients 108, 110, 112). Thus, FIG. 1 is intended as anexample, and not as an architectural limitation for the presentinvention.

FIG. 2 is a schematic illustration of one embodiment of a data center200 for executing the method of the present invention. The data center200 comprises a plurality of application environment modules 201, 202,and 203, one or more resource arbiters 204 and a plurality of resources205, 206, 207, 208 and 209. Each application environment module 201-203is responsible for handling respective demands 213, 214 and 215 (e.g.,requests for information processing services) that may arrive from aparticular customer or set of clients (e.g., clients 108-112 in FIG. 1).Example client types include: online shopping services, online tradingservices, and online auction services.

In order to process client demands 213, 214 or 215, the applicationenvironments 201-203 may utilize the resources 205-209 within the datacenter 200. As each application environment 201-203 is independent fromthe others and provides different services, each application environment201-203 has its own set of resources 205-209 at its disposal, the use ofwhich must be optimized to maintain the appropriate quality of service(QoS) level for the application environment's clients. An arrow from anapplication environment 201-203 to a resource 205-209 denotes that theresource 205-209 is currently in use by the application environment201-203 (e.g., in FIG. 2, resource 205 is currently in use byapplication environment 201). An application environment 201-203 alsomakes use of data or software objects, such as respective Service LevelAgreements (SLAs) 210, 211 and 212 with its clients, in order todetermine its service-level utility function U(S,D). An example SLA210-212 may specify payments to be made by the client based on meanend-to-end response time averaged over, say, a five-minute timeinterval. Additionally the client workload may be divided into a numberof service classes (e.g., Gold, Silver and Bronze), and the SLA 210-212may specify payments based on details of response time characteristicswithin each service class.

Each application environment 201-203 is in further communication withthe resource arbiter module 204. Although the data center 200illustrated in FIG. 2 utilizes only one resource arbiter 204, thoseskilled in the art will appreciate that multiple resource arbiters maybe implemented in the data center 200. The resource arbiter 204 isresponsible for deciding, at any given time while the data center 200 isin operation, which resources 205-209 may be used by which applicationenvironments 201-203. In one embodiment, the application environments201-203 and resource arbiter 204 are software modules consisting ofautonomic elements (e.g., software components that couple conventionalcomputing functionality with additional self-management capabilities),for example written in Java™, and communication between modules 201-203and 204 takes place using standard Java interfaces. The modules 201-203and 204 may run on a single computer or on different computers connectedby a network such as the Internet or a Local Area Network (LAN), e.g.,as depicted in FIG. 1. In the networked case, communication mayadditionally employ standard network communication protocols such asTCP/IP and HTTP, and standard Web interfaces such as OGSA.

FIG. 3 is a flow chart illustrating the method 300 by which the resourcearbiter 204 makes resource allocation decisions. Referringsimultaneously to FIGS. 2 and 3, the method 300 is initialized at block302 and proceeds to block 304, where the method 300 establishes aservice-level utility function U(S, D) for each application environment201-203. In one embodiment, the variable S is a vector thatcharacterizes the multiple performance measures for multiple serviceclasses, and the variable D is a vector that characterizes the demand.The service level utility indicates how much business value U isobtained by the application environment 201, 202 or 203 for variouslevels S of computational system performance, and for a given level D ofdemand 213-215 for computing service.

In one embodiment, the service-level utility function U(S, D) isestablished by the application environment's SLA 210-212. While eachapplication environment's service-level utility may be based ondifferent performance metrics, all of the service-level utilityfunctions U(S, D) share a common scale of valuation.

In block 306, the method 300 transforms the service-level utilityfunction U(S, D) into a resource-level utility function V(R) for eachapplication environment 201-203. The resource level utility indicateshow much business value V is obtained for a given actual or hypotheticalset or amount of resources R (e.g., selected from resources 205-209)allocated to the application environment 201-203. In one embodiment, Ris a vector. For example, the utility information may express a utilitycurve V(m), the utility obtained from being able to use m computeservers, at various values of m ranging from 0 to the total number ofcompute servers within the data center. Additionally if the servers areof different types, the utility information may express the value ofobtaining m servers of type A, n servers of type B, etc. More generallythe utility information may express V({x}), the value of assigning aparticular collection or set {X} of resources 205-209, for various sets{x} ranging over the power set of possible resources 205-209 that couldbe assigned to the application environment 201-203. The utilityinformation may be expressed, for example, in a parameterized functionalform, or it may also be expressed in terms of values at a set ofdiscrete points which may represent a subset or complete set of allpossible resource levels that could be provided.

The transformation may additionally depend on a set of variablesdescribing the application environment's current state (e.g., currentdemand 213-215, system load, throughput or average response time), or ondifferences between a hypothetical resource allocation R and theapplication environment's current resource allocation R* (e.g., in amanner that reflects any costs associated with switching the allocationfrom R* to R, including delays, machine downtime, etc.). In oneembodiment, the resource-level utility function is calculated accordingto the relationV _(i)(R _(i))=U _(i)(S _(i) , D _(i) , R _(i))   (EQN. 1)such that S_(i) εS_(i) (R_(i), D_(i)), where S_(i) (R_(i), D_(i)) is arelation specifying the set of service levels attainable with resourcesR_(i) and demand D_(i). In one embodiment, the relation S_(i) (R_(i),D_(i)) is obtained by standard computer systems modeling techniques(e.g., queuing theory). In another embodiment, the relation S_(i)(R_(i), D_(i)) may instead or additionally be refined by training on acollection of observed system performance data {(S_(t), R_(t), D_(t))}using standard machine learning procedures (e.g., supervised learningmethods employing standard linear or nonlinear function approximators).

In one embodiment, the resource-level utility function V(R) estimatesthe current value of the current state. In another embodiment, theresource-level utility function estimates the expected cumulativediscounted or undiscounted future value starting from the current state.In one embodiment, any one or more of a number of standard methodologiesmay be employed in the process of estimating expected future value,including prediction and forecasting methodologies such as time-seriesprediction methods and machine learning methodologies such asreinforcement learning algorithms (e.g., Q-Learning, Temporal DifferenceLearning, R-Learning or SARSA).

In block 308, the method 300 communicates the respective resource-levelutility functions for each application environment 201-203 to theresource arbiter 204 and aggregates all resource level utilityfunctions. In one embodiment, while the data center 200 is running, fromtime to time each application environment 201-203 communicates to theresource arbiter 204 information regarding its current resource-levelutility function. Said communication may take place either synchronouslyor asynchronously, and may be initiated by the application environments201-203, or may be in response to a prompt or query issued by theresource arbiter 204.

In block 310, the method 300, having received resource-level utilityinformation from each application environment 201-203, combines saidutility information and thereupon decides how to assign each availableresource 205-209 in the data center 200, in a manner that optimizes thetotal utility obtained. In other words, the resource arbiter 204maximizes the sum of the resource-level utilities,$\max\limits_{R\varepsilon R}{\sum\limits_{i}\quad{{V_{i}\left( R_{i} \right)}.}}$Said resource assignment may include the possibility of a nullassignment, (i.e., the resource 205-209 is not assigned to anyapplication environment 201-203) so that the resource 205-209 may bekept in reserve to handle future workload. For example, in the case ofundifferentiated compute servers within the data center 200, theresource arbiter 204 may utilize the most recent utility curves fromeach application environment 201-203 (V₁ (m), V₂ (m) and V₃ (m)respectively), and then compute an integral number of servers (m₁, m₂,m₃) to assign to each application environment 201-203 so as to maximizethe total V₁ (m₁)+V₂ (m₂)+V₃ (m₃). The determination of an allocationthat optimizes total utility will generally be made by executing anoptimization method. In one embodiment, the values (m₁, m₂, m₃) arefound by using standard linear or nonlinear algorithms such as hillclimbing, simulated annealing, linear programming, or mixed-integerprogramming. Additionally, the objective function optimized by theresource arbiter 204 may also include any switching costs that areincurred when a particular resource 205-209 is reallocated from oneapplication environment 201-203 to another. Said switching costs mayinclude, for example, machine downtime and/or other costs related toinstalling or removing data or software from the machine when it isreallocated.

In block 312, the method 300 executes the resource allocation decisioncalculated in block 310, and communicates the resource allocationdecision to the application environments 201-203. In one embodiment,block 312 additionally involves the causation of manipulations oroperations performed upon the resources 205-209, enabling the resources205-209 to be used by the application environments 201-203 to which theresources 205-209 have been assigned, or associated with de-allocating aresource 205-209 from an application environment 201-203 to which theresource 205-209 is no longer assigned.

FIG. 4 is a schematic illustration of the basic operations andfunctionality of one embodiment of an application environment module 401according to the present invention, wherein the application environmentmodule 401 is any of the application environments 201-203 depicted inFIG. 2. In one embodiment, the application environment module 401comprises an autonomic manager element 402, a workload router 403, and asystem performance monitoring element 404. Interactions of theapplication environment 401 with its SLA 410, its client demand 411, itscurrently allocated resources (e.g., compute servers 420, 421, and 422),and with the resource arbiter element 412, are depicted as they were inFIG. 2.

While the application environment 401 is in operation, from time to timeclient demand 411 is received and transmitted to the router 403, whichthereupon sends said demand 411 to one of the assigned compute servers420, 421, or 422, typically based on the use of a routing orload-balancing method. As client jobs are processed, their intermediateand final output are returned to the submitting client. From time totime the performance monitor 404 may observe, request or receiveinformation regarding measures or statistics of the system performanceof the compute servers 420-422, such as CPU/memory usage, averagethroughput, average response time, and average queue depth. Theautonomic manager 402 combines said performance measures withinformation regarding the demand 411, the SLA 610, and the currentlyallocated resources 420-422, to produce an estimated resource-levelutility function.

In one embodiment, said utility function indicates V(m), the value ofbeing allocated an integral quantity m of undifferentiated computeservers, with the value of m ranging from zero to the total number ofservers in the data center (e.g., data center 200 in FIG. 2). From timeto time said utility function is transmitted to the resource arbiter412, possibly in response to a prompt or query sent from the resourcearbiter 412. From time to time said resource arbiter 412 willadditionally transmit to the application environment 401 updatedinformation regarding its set of allocated resources. The updatedinformation indicates, for example, that certain compute servers 420-422are newly available for usage, or that certain compute servers 420-422previously used by the application environment 401 are to bede-allocated and are no longer available for usage.

In another embodiment, the autonomic manager module 402 of FIG. 4further comprises a capability to model the effect of any adjustableoperational parameters the resources 420-422 may have (e.g., maximumqueue depth, buffer pool sizes, etc.) on the observed systemperformance. The autonomic manager 402 further operates to set saidparameters of the resources 420-422, or of the router 403, or otherinternal parameters, to values such that the resulting system-levelutility function optimizes the resource-level utility function.

In another embodiment of the invention, the autonomic manager module 402of FIG. 4 further comprises a capability to model or predict the demandat future times given the observed current demand 411, and a capabilityto model or predict the system performance at future times given thecurrent demand 411, current performance, and future allocated resources,which may be the same or different from the current allocated resources420-422. The autonomic manager 402 then computes a resource-levelutility function indicating the cumulative discounted or undiscountedfuture utility associated with a hypothetical resource allocation madeat the current time. In one embodiment, the predicted demand andpredicted system performance are deterministic predictions at eachfuture time. In another embodiment, the predicted demand and predictedsystem performance are probability distributions over possible levels ofdemand or performance at each future time. In one embodiment, thecumulative future utility is obtained by summation over a finite numberof discrete future time steps. In another embodiment, the cumulativefuture utility is obtained by integration over a continuous future timeinterval.

In another embodiment of the invention, the autonomic manager module 402of FIG. 4 does not explicitly predict future demand or future systemperformance, but instead uses machine learning procedures to estimatecumulative discounted or undiscounted future utility from a temporalsequence of observed data points, each data point consisting of: anobserved demand, an observed system performance, an observed resourceallocation, and an observed payment as specified by the SLA 410. In oneembodiment, the machine learning procedure consists of a standardreinforcement learning procedure such as Q-Learning, Temporal DifferenceLearning, R-Learning or SARSA.

FIG. 5 is a high level block diagram of the present dynamic resourceallocation system that is implemented using a general purpose computingdevice 500. In one embodiment, a general purpose computing device 500comprises a processor 502, a memory 504, a dynamic resource allocator ormodule 505 and various input/output (I/O) devices 506 such as a display,a keyboard, a mouse, a modem, and the like. In one embodiment, at leastone I/O device is a storage device (e.g., a disk drive, an optical diskdrive, a floppy disk drive). It should be understood that the dynamicresource allocator 505 can be implemented as a physical device orsubsystem that is coupled to a processor through a communicationchannel.

Alternatively, the dynamic resource allocator 505 can be represented byone or more software applications (or even a combination of software andhardware, e.g., using Application Specific Integrated Circuits (ASIC)),where the software is loaded from a storage medium (e.g., I/O devices506) and operated by the processor 502 in the memory 504 of the generalpurpose computing device 500. Thus, in one embodiment, the resourceallocator 505 for allocating resources among entities described hereinwith reference to the preceding Figures can be stored on a computerreadable medium or carrier (e.g., RAM, magnetic or optical drive ordiskette, and the like).

The functionalities of the arbiters and the application environmentsdescribed with reference to FIGS. 2 and 4 may be performed by softwaremodules of various types. For example, in one embodiment, the arbitersand/or application environments comprise autonomic elements. In anotherembodiment, the arbiters and/or application environments compriseautonomous agents software as may be constructed, for example, using theAgent Building and Learning Environment (ABLE). The arbiters and/orapplication environments may all run on a single computer, or they mayrun independently on different computers. Communication between thearbiters and the application environments may take place using standardinterfaces and communication protocols. In the case of arbiters andapplication environments running on different computers, standardnetwork interfaces and communication protocols may be employed, such asWeb Services interfaces (e.g., those employed in the Open Grid ServicesArchitecture (OGSA)).

Thus, the present invention represents a significant advancement in thefield of dynamic resource allocation. A method and apparatus areprovided that enable a finite number of resources to be dynamicallyallocated among a number of entities or application environments capableof performing computational work given the resources. The allocation isperformed in a manner that optimizes the business value of theenterprise providing the computing services to a number of clients.

While foregoing is directed to the preferred embodiment of the presentinvention, other and further embodiments of the invention may be devisedwithout departing from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. An automated method for allocating resources among a plurality ofresource-using computational entities in a data processing system, themethod comprising: establishing a service-level utility for each of saidplurality of resource-using entities; and transforming saidservice-level utility into a resource-level utility for each of saidplurality of resource-using entities.
 2. The method of claim 1, whereinthe service-level utility is representative of an amount of businessvalue obtained by each of said plurality of resource-using entities forvarious levels of performance and demand associated with theresource-using entity.
 3. The method of claim 1, wherein theresource-level utility is representative of an amount of business valueobtained by each of said plurality of resource-using entities when aquantity of said resources is allocated to the resource-using entity. 4.The method of claim 1, further comprising the steps of: aggregating saidresource-level utilities of all of said plurality of resource-usingentities; and computing a resource allocation from the aggregatedutility information.
 5. The method of claim 4, further comprising thestep of: executing and conveying to the plurality of resource-usingentities said resource allocation.
 6. The method of claim 1, wherein atleast one of said plurality of resource-using entities operates to setits internal parameters, or an adjustable parameter of the resources theresource-using entity has been allocated so as to optimize theservice-level utility, the resource-level utility, or both.
 7. Themethod of claim 3, wherein the resource-level utility indicates, for atleast one of said plurality of resource-using entities, a currentutility based on current state descriptions of said at least oneresource-using entity.
 8. The method of claim 3, wherein theresource-level utility indicates, for at least one of said plurality ofresource-using entities, an estimated cumulative discounted orundiscounted future utility starting from current state descriptions ofsaid at least one resource-using entity.
 9. The method of claim 8,wherein the estimated cumulative discounted or undiscounted futureutility is based, for at least one of said plurality of resource-usingentities, upon predictions of future state descriptions of said at leastone resource-using entity.
 10. The method of claim 8, wherein theestimated cumulative discounted or undiscounted future utility istrained on a temporal sequence of observed data using an adaptivemachine learning procedure.
 11. The method of claim 10, wherein themachine learning procedure is a reinforcement learning procedure. 12.The method of claim 11, wherein the reinforcement learning procedure isQ-Learning, Temporal Difference Learning, R-Learning or SARSA.
 13. Themethod of claim 8, wherein the estimated cumulative discounted orundiscounted future utility is trained on a temporal sequence ofobserved data using a time-series prediction method.
 14. The method ofclaim 4, wherein the step of aggregating said resource-level utilitiesof all of said plurality of resource-using entities is initiated by saidplurality of resource-using entities.
 15. The method of claim 4, whereinthe step of aggregating said resource-level utilities of all of saidplurality of resource-using entities is initiated by at least oneresource arbiter adapted to compute said resource allocation from theaggregated utility information.
 16. The method of claim 4, wherein thestep of computing a resource allocation from the aggregated utilityinformation comprises executing an optimization method to maximize atotal utility of said data processing system.
 17. The method of claim16, wherein said optimization method comprises a standard linear ornonlinear algorithm.
 18. The method of claim 17, wherein saidoptimization method is hill climbing, simulated annealing, linearprogramming or mixed-integer programming.
 19. The method of claim 4,wherein the step of computing a resource allocation from the aggregatedutility information comprises computing a cost that may be incurred inreallocating at least one of said resources from one of said pluralityof resource-using entities to another.
 20. The method of claim 1,wherein the resource-level utility is a function of client demandreceived by one of said plurality of resource-using entities and of aservice-level agreement governing the performance of said one of saidplurality of resource-using entities.
 21. A computer readable mediumcontaining an executable program for allocating resources among aplurality of resource-using computational entities in a data processingsystem, where the program performs the steps of: establishing aservice-level utility for each of said plurality of resource-usingentities; and transforming said service-level utility into aresource-level utility for each of said plurality of resource-usingentities.
 22. The computer readable medium of claim 21, wherein saidprogram further performs the steps of: aggregating said resource-levelutilities of all of said plurality of resource-using entities; andcomputing a resource allocation from the aggregated utility information.23. The computer readable medium of claim 22, wherein said programfurther performs the step of: executing and conveying to the pluralityof resource-using entities said resource allocation.
 24. The computerreadable medium of claim 21, wherein at least one of said plurality ofresource-using entities operates to set its internal parameters, or anan adjustable parameter of the resources the resource-using entity hasbeen allocated so as to optimize the service-level utility, theresource-level utility, or both.
 25. The computer readable medium ofclaim 21, wherein the resource-level utility indicates an estimatedcumulative discounted or undiscounted future utility starting fromcurrent state descriptions of said plurality of resource-using entities.26. The computer readable medium of claim 21, wherein the resource-levelutility indicates a current utility based on current state descriptionsof said plurality of resource-using entities.
 27. The computer readablemedium of claim 21, wherein the resource-level utility indicates anestimated cumulative discounted or undiscounted future utility startingfrom current state descriptions of said plurality of resource-usingentities.
 28. The computer readable medium of claim 23, wherein the stepof computing a resource allocation from the aggregated utilityinformation comprises executing an optimization algorithm to maximize abusiness value of said data processing system.
 29. A data processingsystem, comprising: a plurality of entities adapted for processingclient demands; a plurality of resources adapted for allocation to saidplurality of entities; and at least one resource arbiter adapted forallocating said plurality of resources among said plurality of entitiesin a manner that optimizes a business value of the data processingsystem.
 30. The data processing system of claim 25, wherein saidplurality of entities are further adapted for transforming a respectiveservice-level utility function into a corresponding service-levelutility function.
 31. The data processing system of claim 25, whereinsaid plurality of entities and said at least one resource arbiter arerun on a single computer.
 32. The data processing system of claim 25,wherein said plurality of entities and said at least one resourcearbiter are run on different computers connected by a network.
 33. Thedata processing system of claim 25, wherein said plurality of entitiesand said at least one resource arbiter are software modules comprisingautonomic elements.
 34. The data processing system of claim 25, whereinthe data processing system is a server, a client computer or a network.